GSP-006 Monitoring Cloud Infrastructure with Stackdriver

2017년 10월 23일 월요일

오전 9:40

링크: https://qwiklabs.com/focuses/6476

 

오버뷰

Stackdriver Monitoring는

 

- Dashboard, Alert 제공

- 클라우드 서비스, VM, 오픈소스 서버들(MongoDB, Apache, Nginx, Elasticsearch 등등) 의 Performance mecrics 볼 수 있음

- Stackdriver Monitoring Console 을 이용해 설정가능

 

할일들

- Stackdriver Monitoring 을 enable

- Google Stackdriver Monitoring 컨셉을 연습으로 익혀보기

- Stackdriver Monitoring 을 둘러보기

 

* 예전에 빅쿼리 로딩 작업의 히스토리가

- 웹콘솔을 통해 로딩하면 보이는데, bq CLI로 로딩하면 안보였던 이슈가 있었는데

- 누군가가 Stackdriver 로도 안보이냐고 물어봤었다.

40521b5cbd41bb11.png

 

 

모니터 할 게 있어야 모니터하지

 

모니터할 리소스부터 만들어보자

1) VM instance

2) Cloud SQL instance

 

Virtual Machine instance만들기

 

- 많이 만들어보았음.

- 웹콘솔에서 Compute Engine 가서 그냥 만들면 됨. 끝

qwiklabs-gcp4df3f61fb1 • Create an instance Q Google Cloud Platform Compute Engine VM instances Instance groups Instance templates Disks Snapshots Images Committed use discounts Metadata Health checks Zones Operations Quotas Settings Name instance-I Zone us-centrall-a Machine type 1 vCPU Boot disk 3.75 GB memory New 10 GB standard persistent disk Image Debian GNU/Linux g (stretch) Customize Change Identity and API access Service account Compute Engine default service account Access scopes • Allow default access Allow full access to all Cloud APIs Set access for each API Firewall Add tags and firewall pules to allow specific network traffic from the Internet Allow HTTPtratfic Allow HTTPS tramc Management, disks, networking, SSH keys The following options have been customized: Network interfaces You will be billed for this instance Learn more Cancel

 생성확인

qwiklabs-gcp4df3f61fb1 • VM instances Filter VM instances Q Google Cloud Platform Compute Engine VM instances Instance groups Instance templates Disks Snapshots SHOW INFO PANEL Name A e Instance-I Zone us-centrall-a Recommendation Internal IP 10.1280.2 External IP Connect

 

 

Cloud SQL Instance 만들기

 

- 웹콘솔에서  SQL 가서 그냥 만들면 됨. 끝

Google Cloud Platform qwiklabs-gcp-4df3f61fb1- • Create an instance SQL Choose a database engine . MySQL Versions 5.6 or SD PostgreSQ Version 98 Next

 

Google Cloud Platform qwiklabs-gcp-4df3f61fb1- • k- Choose a MySQL instance type SQL There are two types of Cloud SQL MySQL instances Learn more MySQL Second Generation (Recommended) High performance, high storage capacity, low cost. • Up to 7X throughput and 20X storage capacity of First Generation • Less expensive than First Generation for most use cases • Option to add High Availability failover and read replication • Configurable backup period and maintenance window • Supports only MySQL 56 and 5.7 Choose Second Generation MySQL First Generation (Legacy) Older version of Cloud SQL providing basic performance and storage capacity. Does not support MySQL 5.7. Choose First Generation

 

Google Cloud Platform qwiklabs-gcp-4df3f61fbE • SQL nstance 10 k- Create a MySQL Second Generation instance Cannot be changed I er_ Use lowercase letters, numbers, art hypheny Start with a letter. my-mysq Edb Root password Set a password for lhe root user Learn more gsutjr No password Location For Setter performance, keep your data close to the services that need it Generate Region us-centrall Show configuration options Zone Crea te Cancel

 

 

 

생성확인

Google Cloud Platform qwiklabs-gcp4df3f61fb1 • CREATE INSTANCE SQL Filter instances Instance 10 'E) O my-mysql-db Instances Type MySQL 2nd Gen 37 High availability SHOW INI Storage used IP address 3520280.7 Instance connection name qwiklabs-gcp- 4df3f61fb169cc17:us- centrall :my-mysql-db Location us-central I-a

 

Stackdriver Monitoring 을 Enable 하자

 

- 어라~ 30일만 무료사용 가능?

- AWS도 모니터링 할 수 있다.

 

Google Cloud Platform Home Pins appear here Network services Interconnect STACKORIVER Monitoring Debug x

Stackdriver Select account Create your free Stackdriver account Start your 30 day free trial. No credit card required. Select or create a new Google Cloud Platform project to store your Stackdriver account settings and user permissions. The selection cannot be changed, but you can create other Stackdriver accounts later Learn more Google Cloud Platform project qwikIabs-gcpudfSf61 fbl 6gcc17 Cancel x

 

모니터링, 로깅 agent를 설치하면 좋다고 한다.

- Google Cloud Shell 이나 SDK를 깐 local PC에 설치하면 될듯

- 이 랩에서는 일단 스킵해도 된다

 

시스템 생성 대체 텍스트: Install the Stackdriver Agents recommended Get the most 0니t Of your free Stackdriver account by installing the Stackdriver Monitoring and Logging agents on each Of your VM instances. Agents collect more information from your VM instances, including metrics and logs from third party applications 1 - Switch to the terminal connected to your 1,等』 instance, or create 3 new one. 2 Install the Stackdriver agents by running the following commands on your instance: # TO Install the Stackdrive『 Mlitoring agent 5 curl -5剄 https://repo.stackdriver.*. Sh 5 5血 bash Stack—install. Sh --write—gcm # TO install the Stackdrive『 logging agent• curl -5剄 5 5血 bash Sh For more details and troubleshoot options when installing the agents, see Installing the Stackdriver Monitoring Agent and Installing the Stackdriver Logging Agent. Conünue

   

 

Finished initial collection! Launch monitoring Create an alerting policy Define alerting rules to notify you when something goes wrong. Receive notifications via email, SMS, PagerDuty, HipChat, and more. Alert on individual metrics and thresholds or on aggregate group performance. Learn more about alerts. Set up and monitor uptime checks Configure an uptime check to monitor the health of any Internet-facing resource: a web page, instance, or load balancer. Well check the availability of your resources from probe locations around the world. Learn more about uptime checks. View release notes Check out release notes to see what we've been up to lately.

 

 

일단 딱 VM 과 SQL이 보인다

Instance (GCE) CPU Usage (GCE Monitoring) Google Cloud SQL - Queries Executed Oct 23, 2017 10:04 AM • instance-I 10:15 Name instance-I 0.064 10 30 10:45 0.02 Value 6.42% 10:15 10 30 10:45 Name my mysql db Value o_38/s

 

 

Monitoring Dashboard

 

Uptime Checks

 

* 현재는 텅텅 비어있음

- 다른 코드랩에서 setup 해주고 좀 더 볼 예정임

 

웹페이지, 인스턴스, 리소스 그룹의 건강상태를 빠르게 검증해준다.

각각의 check 설정은 전세계 다양한 위치에서 정기적으로 접촉된다

Uptime check 는 Alert policy에서의 컨디션으로 사용될 수 있다.

 

Uptime Checks O Create Check You dont have any uptime checks Learn more about the uptime monitoring functionality. Create an Uptime Check

New Uptime Check Title Check Title Check Type H TTp Resource Type URL Hostname examplecom path /index_html Select protocol and enter the path component of the LJRL you want to check Check every 5 minutes Advanced Options Test Cancel

 

 

Incidents

 

Alerting policy를 적용해서 문제발생을 알아채자

- Stackdriver에서 제공하는 다양한 metrics 와 health check를 이용하여 policy에 이용하자

 

1) Alert policy에 걸리면 Incident 가 생성되며

2) 이것은 Incident 섹션에서 볼 수 있다

3) Incident 를 처리하고 나서 Closed 처리할 수 있다.

 

* 여기서는 Incident 를 따로 다루진 않을거다

Incidents O OPEN (0) ACKNOWLEDGED (0) No open incidents Create Alerting Policy

 

Alerting / Policies / Create Create new alerting policy 2 3 4 Conditions Conditions describe when apps and services are considered unhealthy. When conditions are met, they trigger alerting policy violations. Learn more ? + Add Condition (optional) Notifications When alerting policy violations occur, you will be notified via these channels. Learn more + Add Notification (optional) Documentation When email notifications are sent, they'll include any text entered here. This can convey useful information about the problem and ways to approach fixing it. + Add Documentation Name this policy A policy's name is used in identifying which policies were triggered, as well as managing configurations of different policies. enter a policy name

 

 

Event Log

 

프로젝트에서 발생한 이벤트들이 쌓인다

 

- 서버가 생성되었다.

- 서버가 restart 되었다

- 나만의 메시지를 이벤트로 추가할 수 있다.

 

그냥 로그 함 봄

 

Google Cloud Platform Stackdriver Logging Logs Logs-based metrics Exports Resource usage qwiklabs-gcp4df3f61fb1 • CREATE METRIC CREATE EXPORT ; You have 30 days left in your Stackdriver Premium trial. We will enforce a 7-clay retention for logs In the Basic Tier. Please upgrade to the Premium Tier to keep your logs with 30-day retention. Learn more Filter by label or text search GCE VM Instance, instance-I 2017-10-23 JS All logs Any log level Jump to date No older entries found matching current filter. .478 42 . 454 48.824 48 .905 Engine insert us-centrall-a: instance-I googleIø9956-student@qÆkIabs.net Compute compute. instances. insert {"ip_address" : {"up I" :"https://www.googIeapis.com/compute/beta/projects/gwikIabs-gcp-4df3f61fb169cc17/zones/us-centraII-a/ins Engine insert us-centrall-a:instance-l googleIø9956-student@qÆkIabs.net Compute compute . instances . insert • . net"}, "resource" : {"type": "instance" , "zone": "us-cer

 

Google Cloud Platform Stackdriver Logging Logs Logs-based metrics Exports Resource usage qwiklabs-gcp4df3f61fb1 • CREATE METRIC CREATE EXPORT ; You have 30 days left in your Stackdriver Premium trial. We will enforce a 7-clay retention for logs In the Basic Tier. Please upgrade to the Premium Tier to keep your logs with 30-d Filter by label or text search Cloud SQL Database, qwiklabs-gcp-4dt3t61t_ 2017-10-23 JS All logs Any log level Jump to date la: 51: ee .669 2017-le-23Ta1 : 51 : ea. 6691562 la: 51 : ee. 669 2017-le-23Ta1 : 51 : ea. 6698562 la: 51: ee .721 2017-1e-23Ta1 : 51 : ea. 7211842 la: 51 ee. 722 2017-le-23Ta1 : 51 : ea. la: 51: ee .726 2017-le-23Ta1 : 51 : ea. la: 51 :ee. 726 2017-le-23Ta1 : 51 : ea. 7250832 la: 51 .758 2017-le-23Ta1 : 51 : ea. 7580122 la: 51 ee. 766 2017-le-23Ta1 : 51 : ea. 7664772 la: 51 .767 2017-1e-23Ta1 : 51 : ea. 7665782 la: 51 ee. 769 2017-le-23Ta1 : 51 : ea. 7691692 la: 51 .77a 2017-le-23Ta1 : 51 : ea. 7701322 la: 51: ee .77a 2017-le-23Ta1 : 51 : ea. 7702352 la: 51: ee. 771 2017-le-23Ta1 : 51 : ea. la: 51: ee .771 2017-1e-23Ta1 : 51 : ea. 7711742 la: 51: ee. 771 2017-le-23Ta1 : 51 : ea. 7711872 la: 51: ee .771 2017-le-23Ta1 : 51 : ea. 7711982 la: 51: ee. 775 2017-le-23Ta1 : 51 : ea. e [Note] e [Note] e [Note] e [Note] e [Note] e [Note] e [Note] e [Note] e [Note] e [Note] InnoDB : InnoDB : InnoDB : InnoDB : 32 non-redo rollback segment(s) are active. Waiting for purge to start 5.7.14 started; log sequence number 113749864 Loading buffer pool(s) from Semi-sync replication initialized for transactions. Semi-sync replication enabled on the master . Starting ack receiver thread Found gtids in binlog: Found gtids purged from binlog: Skipping generation of SSL certificates as options related to SSL are specified. e [Idarning] CA certificate . pem is self signed. e [Note] Skipping generation of RSA key pair as key files are present in data directory . e [Note] Server hostname (bind-address) e [Note] IPv6 is available. port: 33136 e [Note] resolves to e [Note] Server socket created on IR: ' e [Idarning] ' user' entry 'root@locelhost' ignored in - -skip-name-resolve mode.

 

Graphs and Charts

 

오른편에는 다양한 그래프와 차트를 제공한다

 

Compute Engine Instance Monitoring Data

 

Stackdriver 에서 Resource > instances 에서 원하는 Compute Engine VM instance를 선택

 

- Uptime Chekcs

- Event Log

- Ip Addresses, Launch time, Volume 등등

- Graph - CPU Usage, Disk I/O, Network Traffic

 

Stackdriver qwik1abs-gcp-4df3f61fb169cc17 Monitoring Overview Resources Alerting Uptime Checks Groups Dashboards qwiklabs-gcp-4df3f61fb169cc17 Metrics Explorer INFRASTRUCTURE Block Storage Volumes Instances Security Groups GCP Cloud SQL

 

Stackdriver qwik1abs-gcp-4df3f61fb169cc17 Monitoring Overview Resources Alerting Uptime Checks Groups Dashboards Infrastructure / Instances O Filter m HEALTH NAME instance-I Showing 1 1 of 1 instance (1 running) ZONE gce:us-centrall-a

 

 

VM 모니터링

Infrastructure / Instances / instance-I TIME lh 6h Id lw 1m 6w Oct 23, 11:07 am +09001 custom Incidents o OPEN (0) ACKNOWLEDGED (0) No open incidents Uptime Checks o Create Alerting Policy RESOLVED (0) Create Check oct23, 10:07 am 10:15 10 30 10:45 11 AM You have no uptime checks for this resource. Events Overview DETAILS Id State Events Tags public IP Private IP Location Instance Type Launch time Attached Volumes 30240011 17202741034 running Name = instance-I 35.194.35.138 10.128.0.2 gce:us-centrall-a n I-standard-I oct 23, 2017 1045 am UTC+OGOO instance-I (10 G8) CPU Usage Disk I/O Network Traffic 0.02 1.000,000 800.000 600.000 400.000 200.000

 

 

Cloud SQL Monitoring Data

 

Resource > Cloud SQL

Inventory (목록) 에서 원하는 DB 선택

 

- Incident

- Event Log

- Settings, IP Configuration, Backup Configuration

 

Services / CloudSQL / my-mysql-db TIME lh 6h Id lw 1m 6w custom Incidents o OPEN (0) ACKNOWLEDGED (0) No open incidents Events Cloud SQL Details Create Alerting Policy RESOLVED (0) oct23, 10:09 am 10:15 Queries Oct 23, 11:09 am [UTC +09001 10 30 10:45 11 AM Name State Region DB Version Max Disk Size Actions my-mysql-db RUNNA8LE us-central 0.0 View in Google Cloud Console e Settings Tier Replication Type Activation Policy Authorized GAE Apps IP Configuration Enabled db-nl-standard-l SYNCHRONOUS ALWAYS true InnoDB Pages Read/Write Network Connections

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Microsoft OneNote 2016에서 작성